Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.62%. Comparing base (dd3153d) to head (bc55c64).
⚠️ Report is 1 commits behind head on main.

@@            Coverage Diff             @@
##             main     #175      +/-   ##
==========================================
+ Coverage   93.58%   93.62%   +0.04%     
==========================================
  Files          17       17              
  Lines         561      596      +35     
==========================================
+ Hits          525      558      +33     
- Misses         36       38       +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Logic looks mostly good here, but I have some questions to make sure I understand what's going on.

1. What are the changes being made to the azure_scripts/performance.ipynb notebook? From what I can tell, it looks like cell formatting and some cell output (which we don't want in the github repo anyway), but just wondering if there's been any logic changes. This is the notebook we use to compute our performance stats right now so I want to make sure it doesn't break. It doesn't look like there's any of the valueset accuracy stuff added to it, so if we want to document that as a second notebook that might be my preference?
2. Out results on most of these tests look absolutely terrible, which is giving me a bit of concern. We were hitting 65% accuracy on our earlier tests, but here using the same test case generation methodology, I can see 8 total cases where we find the right answer out of the hundreds/thousands you're running. That seems like something deeper is wrong to me. How is eval_results_snipped_with_loinc_codes.txt being generated? Are you getting these results using the regular performance notebook, or the copy you have in Azure? I feel like something has to be misaligned somewhere because even if we're not getting the right valuesets when the model is wrong, the rate we're getting regular codes flat-out wrong suggests something off to me.

1. The primary changes are: 1) changing from pickle to jsonl, and in the final step and 2) outputting the results of each run into jsonls in the local folder. I don't think it introduces any change to how performance stats are calculated. I'm not opposed to breaking this out into another notebook to avoid multiple uses/updates being tied to one notebook.
2. So far, I've only run this against the JSONLs generated back in December that you flagged an issue with. So I'm not sure if the accuracy would be impacted by that issue. I can re-review the notebook to see if there's anything wrong with how that loop through is going.

@robertandremitchell Yeah let's split it out into a second notebook. JSONL is super slow for general performance so this'll save us the hassle of switching back and forth when we want to do fine-tuning runs. Also, before uploading the new notebook, can you hit the Clear All Outputs option so we don't commit stuff like the pip install log? Should be accessible from the three dots at the top left of the notebook editing window in Azure.

As far as the accuracy goes, I don't think the performance should really be affected that much by those nulls--I just don't understand why we can hit 65+% on our regular runs with the validation set but then for you it's showing up as minimal. Especially because last week I sanity checked the JSONL files by doing a performance run where I loaded from JSONL and then still achieved our expected performance numbers. Can you ping me when your split-out notebook is updated in Azure? I think looking at it there might help me see if something is up.